{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "60bf2cbb",
   "metadata": {},
   "source": [
    "# 3rd party integration - RayTune, Weights & Biases\n",
    "\n",
    "This notebook provides guideline for integration of external library functions in the model training process through `Callback` objects, a popular concept of using objects as arguments for other objects.\n",
    "\n",
    "**[DISCLAIMER]**\n",
    "\n",
    "We show integration of RayTune (a hyperparameter tuning framework) and Weights & Biases (ML projects experiment tracking and versioning solution) in the `pytorch_widedeep` model training process. We did not include `RayTuneReporter` and `WnBReportBest` in the library code to minimize the dependencies on other libraries that are not directly included in the model design and training."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9822d48a",
   "metadata": {},
   "source": [
    "## Initial imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d073f793",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from typing import Optional, Dict\n",
    "import os\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import wandb\n",
    "from torch.optim import SGD, lr_scheduler\n",
    "\n",
    "from pytorch_widedeep import Trainer\n",
    "from pytorch_widedeep.preprocessing import TabPreprocessor\n",
    "from pytorch_widedeep.models import TabMlp, WideDeep\n",
    "from torchmetrics import F1Score as F1_torchmetrics\n",
    "from torchmetrics import Accuracy as Accuracy_torchmetrics\n",
    "from torchmetrics import Precision as Precision_torchmetrics\n",
    "from torchmetrics import Recall as Recall_torchmetrics\n",
    "from pytorch_widedeep.metrics import Accuracy, Recall, Precision, F1Score, R2Score\n",
    "from pytorch_widedeep.initializers import XavierNormal\n",
    "from pytorch_widedeep.callbacks import (\n",
    "    EarlyStopping,\n",
    "    ModelCheckpoint,\n",
    "    Callback,\n",
    ")\n",
    "from pytorch_widedeep.datasets import load_bio_kdd04\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n",
    "\n",
    "from ray import tune\n",
    "from ray.tune.schedulers import AsyncHyperBandScheduler\n",
    "from ray.tune import JupyterNotebookReporter\n",
    "from ray.air.integrations.wandb import WandbLoggerCallback\n",
    "\n",
    "# from ray.tune.integration.wandb import wandb_mixin\n",
    "\n",
    "import tracemalloc\n",
    "\n",
    "tracemalloc.start()\n",
    "\n",
    "# increase displayed columns in jupyter notebook\n",
    "pd.set_option(\"display.max_columns\", 200)\n",
    "pd.set_option(\"display.max_rows\", 300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3157bb9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "class RayTuneReporter(Callback):\n",
    "    r\"\"\"Callback that allows reporting history and lr_history values to RayTune\n",
    "    during Hyperparameter tuning\n",
    "\n",
    "    Callbacks are passed as input parameters to the ``Trainer`` class. See\n",
    "    :class:`pytorch_widedeep.trainer.Trainer`\n",
    "\n",
    "    For examples see the examples folder at:\n",
    "\n",
    "        .. code-block:: bash\n",
    "\n",
    "            /examples/12_HyperParameter_tuning_w_RayTune.ipynb\n",
    "    \"\"\"\n",
    "\n",
    "    def on_epoch_end(\n",
    "        self, epoch: int, logs: Optional[Dict] = None, metric: Optional[float] = None\n",
    "    ):\n",
    "        report_dict = {}\n",
    "        for k, v in self.trainer.history.items():\n",
    "            report_dict.update({k: v[-1]})\n",
    "        if hasattr(self.trainer, \"lr_history\"):\n",
    "            for k, v in self.trainer.lr_history.items():\n",
    "                report_dict.update({k: v[-1]})\n",
    "        tune.report(report_dict)\n",
    "\n",
    "\n",
    "class WnBReportBest(Callback):\n",
    "    r\"\"\"Callback that allows reporting best performance of a run to WnB\n",
    "    during Hyperparameter tuning. It is an adjusted pytorch_widedeep.callbacks.ModelCheckpoint\n",
    "    with added WnB and removed checkpoint saving.\n",
    "\n",
    "    Callbacks are passed as input parameters to the ``Trainer`` class.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    wb: obj\n",
    "        Weights&Biases API interface to report single best result usable for\n",
    "        comparisson of multiple paramater combinations by, for example,\n",
    "        `parallel coordinates\n",
    "        <https://docs.wandb.ai/ref/app/features/panels/parallel-coordinates>`_.\n",
    "        E.g W&B summary report `wandb.run.summary[\"best\"]`.\n",
    "    monitor: str, default=\"loss\"\n",
    "        quantity to monitor. Typically `'val_loss'` or metric name\n",
    "        (e.g. `'val_acc'`)\n",
    "    mode: str, default=\"auto\"\n",
    "        If ``save_best_only=True``, the decision to overwrite the current save\n",
    "        file is made based on either the maximization or the minimization of\n",
    "        the monitored quantity. For `'acc'`, this should be `'max'`, for\n",
    "        `'loss'` this should be `'min'`, etc. In `'auto'` mode, the\n",
    "        direction is automatically inferred from the name of the monitored\n",
    "        quantity.\n",
    "\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(\n",
    "        self,\n",
    "        wb: object,\n",
    "        monitor: str = \"val_loss\",\n",
    "        mode: str = \"auto\",\n",
    "    ):\n",
    "        super(WnBReportBest, self).__init__()\n",
    "\n",
    "        self.monitor = monitor\n",
    "        self.mode = mode\n",
    "        self.wb = wb\n",
    "\n",
    "        if self.mode not in [\"auto\", \"min\", \"max\"]:\n",
    "            warnings.warn(\n",
    "                \"WnBReportBest mode %s is unknown, \"\n",
    "                \"fallback to auto mode.\" % (self.mode),\n",
    "                RuntimeWarning,\n",
    "            )\n",
    "            self.mode = \"auto\"\n",
    "        if self.mode == \"min\":\n",
    "            self.monitor_op = np.less\n",
    "            self.best = np.Inf\n",
    "        elif self.mode == \"max\":\n",
    "            self.monitor_op = np.greater  # type: ignore[assignment]\n",
    "            self.best = -np.Inf\n",
    "        else:\n",
    "            if self._is_metric(self.monitor):\n",
    "                self.monitor_op = np.greater  # type: ignore[assignment]\n",
    "                self.best = -np.Inf\n",
    "            else:\n",
    "                self.monitor_op = np.less\n",
    "                self.best = np.Inf\n",
    "\n",
    "    def on_epoch_end(  # noqa: C901\n",
    "        self, epoch: int, logs: Optional[Dict] = None, metric: Optional[float] = None\n",
    "    ):\n",
    "        logs = logs or {}\n",
    "        current = logs.get(self.monitor)\n",
    "        if current is not None:\n",
    "            if self.monitor_op(current, self.best):\n",
    "                self.wb.run.summary[\"best\"] = current  # type: ignore[attr-defined]\n",
    "                self.best = current\n",
    "                self.best_epoch = epoch\n",
    "\n",
    "    @staticmethod\n",
    "    def _is_metric(monitor: str):\n",
    "        \"copied from pytorch_widedeep.callbacks\"\n",
    "        if any([s in monitor for s in [\"acc\", \"prec\", \"rec\", \"fscore\", \"f1\", \"f2\"]]):\n",
    "            return True\n",
    "        else:\n",
    "            return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6f0ee187",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EXAMPLE_ID</th>\n",
       "      <th>BLOCK_ID</th>\n",
       "      <th>target</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "      <th>40</th>\n",
       "      <th>41</th>\n",
       "      <th>42</th>\n",
       "      <th>43</th>\n",
       "      <th>44</th>\n",
       "      <th>45</th>\n",
       "      <th>46</th>\n",
       "      <th>47</th>\n",
       "      <th>48</th>\n",
       "      <th>49</th>\n",
       "      <th>50</th>\n",
       "      <th>51</th>\n",
       "      <th>52</th>\n",
       "      <th>53</th>\n",
       "      <th>54</th>\n",
       "      <th>55</th>\n",
       "      <th>56</th>\n",
       "      <th>57</th>\n",
       "      <th>58</th>\n",
       "      <th>59</th>\n",
       "      <th>60</th>\n",
       "      <th>61</th>\n",
       "      <th>62</th>\n",
       "      <th>63</th>\n",
       "      <th>64</th>\n",
       "      <th>65</th>\n",
       "      <th>66</th>\n",
       "      <th>67</th>\n",
       "      <th>68</th>\n",
       "      <th>69</th>\n",
       "      <th>70</th>\n",
       "      <th>71</th>\n",
       "      <th>72</th>\n",
       "      <th>73</th>\n",
       "      <th>74</th>\n",
       "      <th>75</th>\n",
       "      <th>76</th>\n",
       "      <th>77</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>279</td>\n",
       "      <td>261532</td>\n",
       "      <td>0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>32.69</td>\n",
       "      <td>0.30</td>\n",
       "      <td>2.5</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1256.8</td>\n",
       "      <td>-0.89</td>\n",
       "      <td>0.33</td>\n",
       "      <td>11.0</td>\n",
       "      <td>-55.0</td>\n",
       "      <td>267.2</td>\n",
       "      <td>0.52</td>\n",
       "      <td>0.05</td>\n",
       "      <td>-2.36</td>\n",
       "      <td>49.6</td>\n",
       "      <td>252.0</td>\n",
       "      <td>0.43</td>\n",
       "      <td>1.16</td>\n",
       "      <td>-2.06</td>\n",
       "      <td>-33.0</td>\n",
       "      <td>-123.2</td>\n",
       "      <td>1.60</td>\n",
       "      <td>-0.49</td>\n",
       "      <td>-6.06</td>\n",
       "      <td>65.0</td>\n",
       "      <td>296.1</td>\n",
       "      <td>-0.28</td>\n",
       "      <td>-0.26</td>\n",
       "      <td>-3.83</td>\n",
       "      <td>-22.6</td>\n",
       "      <td>-170.0</td>\n",
       "      <td>3.06</td>\n",
       "      <td>-1.05</td>\n",
       "      <td>-3.29</td>\n",
       "      <td>22.9</td>\n",
       "      <td>286.3</td>\n",
       "      <td>0.12</td>\n",
       "      <td>2.58</td>\n",
       "      <td>4.08</td>\n",
       "      <td>-33.0</td>\n",
       "      <td>-178.9</td>\n",
       "      <td>1.88</td>\n",
       "      <td>0.53</td>\n",
       "      <td>-7.0</td>\n",
       "      <td>-44.0</td>\n",
       "      <td>1987.0</td>\n",
       "      <td>-5.41</td>\n",
       "      <td>0.95</td>\n",
       "      <td>-4.0</td>\n",
       "      <td>-57.0</td>\n",
       "      <td>722.9</td>\n",
       "      <td>-3.26</td>\n",
       "      <td>-0.55</td>\n",
       "      <td>-7.5</td>\n",
       "      <td>125.5</td>\n",
       "      <td>1547.2</td>\n",
       "      <td>-0.36</td>\n",
       "      <td>1.12</td>\n",
       "      <td>9.0</td>\n",
       "      <td>-37.0</td>\n",
       "      <td>72.5</td>\n",
       "      <td>0.47</td>\n",
       "      <td>0.74</td>\n",
       "      <td>-11.0</td>\n",
       "      <td>-8.0</td>\n",
       "      <td>1595.1</td>\n",
       "      <td>-1.64</td>\n",
       "      <td>2.83</td>\n",
       "      <td>-2.0</td>\n",
       "      <td>-50.0</td>\n",
       "      <td>445.2</td>\n",
       "      <td>-0.35</td>\n",
       "      <td>0.26</td>\n",
       "      <td>0.76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>279</td>\n",
       "      <td>261533</td>\n",
       "      <td>0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>33.33</td>\n",
       "      <td>0.00</td>\n",
       "      <td>16.5</td>\n",
       "      <td>9.5</td>\n",
       "      <td>608.1</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.07</td>\n",
       "      <td>20.5</td>\n",
       "      <td>-52.5</td>\n",
       "      <td>521.6</td>\n",
       "      <td>-1.08</td>\n",
       "      <td>0.58</td>\n",
       "      <td>-0.02</td>\n",
       "      <td>-3.2</td>\n",
       "      <td>103.6</td>\n",
       "      <td>-0.95</td>\n",
       "      <td>0.23</td>\n",
       "      <td>-2.87</td>\n",
       "      <td>-25.9</td>\n",
       "      <td>-52.2</td>\n",
       "      <td>-0.21</td>\n",
       "      <td>0.87</td>\n",
       "      <td>-1.81</td>\n",
       "      <td>10.4</td>\n",
       "      <td>62.0</td>\n",
       "      <td>-0.28</td>\n",
       "      <td>-0.04</td>\n",
       "      <td>1.48</td>\n",
       "      <td>-17.6</td>\n",
       "      <td>-198.3</td>\n",
       "      <td>3.43</td>\n",
       "      <td>2.84</td>\n",
       "      <td>5.87</td>\n",
       "      <td>-16.9</td>\n",
       "      <td>72.6</td>\n",
       "      <td>-0.31</td>\n",
       "      <td>2.79</td>\n",
       "      <td>2.71</td>\n",
       "      <td>-33.5</td>\n",
       "      <td>-11.6</td>\n",
       "      <td>-1.11</td>\n",
       "      <td>4.01</td>\n",
       "      <td>5.0</td>\n",
       "      <td>-57.0</td>\n",
       "      <td>666.3</td>\n",
       "      <td>1.13</td>\n",
       "      <td>4.38</td>\n",
       "      <td>5.0</td>\n",
       "      <td>-64.0</td>\n",
       "      <td>39.3</td>\n",
       "      <td>1.07</td>\n",
       "      <td>-0.16</td>\n",
       "      <td>32.5</td>\n",
       "      <td>100.0</td>\n",
       "      <td>1893.7</td>\n",
       "      <td>-2.80</td>\n",
       "      <td>-0.22</td>\n",
       "      <td>2.5</td>\n",
       "      <td>-28.5</td>\n",
       "      <td>45.0</td>\n",
       "      <td>0.58</td>\n",
       "      <td>0.41</td>\n",
       "      <td>-19.0</td>\n",
       "      <td>-6.0</td>\n",
       "      <td>762.9</td>\n",
       "      <td>0.29</td>\n",
       "      <td>0.82</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>-35.0</td>\n",
       "      <td>140.3</td>\n",
       "      <td>1.16</td>\n",
       "      <td>0.39</td>\n",
       "      <td>0.73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>279</td>\n",
       "      <td>261534</td>\n",
       "      <td>0</td>\n",
       "      <td>77.0</td>\n",
       "      <td>27.27</td>\n",
       "      <td>-0.91</td>\n",
       "      <td>6.0</td>\n",
       "      <td>58.5</td>\n",
       "      <td>1623.6</td>\n",
       "      <td>-1.40</td>\n",
       "      <td>0.02</td>\n",
       "      <td>-6.5</td>\n",
       "      <td>-48.0</td>\n",
       "      <td>621.0</td>\n",
       "      <td>-1.20</td>\n",
       "      <td>0.14</td>\n",
       "      <td>-0.20</td>\n",
       "      <td>73.6</td>\n",
       "      <td>609.1</td>\n",
       "      <td>-0.44</td>\n",
       "      <td>-0.58</td>\n",
       "      <td>-0.04</td>\n",
       "      <td>-23.0</td>\n",
       "      <td>-27.4</td>\n",
       "      <td>-0.72</td>\n",
       "      <td>-1.04</td>\n",
       "      <td>-1.09</td>\n",
       "      <td>91.1</td>\n",
       "      <td>635.6</td>\n",
       "      <td>-0.88</td>\n",
       "      <td>0.24</td>\n",
       "      <td>0.59</td>\n",
       "      <td>-18.7</td>\n",
       "      <td>-7.2</td>\n",
       "      <td>-0.60</td>\n",
       "      <td>-2.82</td>\n",
       "      <td>-0.71</td>\n",
       "      <td>52.4</td>\n",
       "      <td>504.1</td>\n",
       "      <td>0.89</td>\n",
       "      <td>-0.67</td>\n",
       "      <td>-9.30</td>\n",
       "      <td>-20.8</td>\n",
       "      <td>-25.7</td>\n",
       "      <td>-0.77</td>\n",
       "      <td>-0.85</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-20.0</td>\n",
       "      <td>2259.0</td>\n",
       "      <td>-0.94</td>\n",
       "      <td>1.15</td>\n",
       "      <td>-4.0</td>\n",
       "      <td>-44.0</td>\n",
       "      <td>-22.7</td>\n",
       "      <td>0.94</td>\n",
       "      <td>-0.98</td>\n",
       "      <td>-19.0</td>\n",
       "      <td>105.0</td>\n",
       "      <td>1267.9</td>\n",
       "      <td>1.03</td>\n",
       "      <td>1.27</td>\n",
       "      <td>11.0</td>\n",
       "      <td>-39.5</td>\n",
       "      <td>82.3</td>\n",
       "      <td>0.47</td>\n",
       "      <td>-0.19</td>\n",
       "      <td>-10.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1491.8</td>\n",
       "      <td>0.32</td>\n",
       "      <td>-1.29</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-34.0</td>\n",
       "      <td>658.2</td>\n",
       "      <td>-0.76</td>\n",
       "      <td>0.26</td>\n",
       "      <td>0.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>279</td>\n",
       "      <td>261535</td>\n",
       "      <td>0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>27.91</td>\n",
       "      <td>-0.35</td>\n",
       "      <td>3.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>1921.6</td>\n",
       "      <td>-1.36</td>\n",
       "      <td>-0.47</td>\n",
       "      <td>-32.0</td>\n",
       "      <td>-51.5</td>\n",
       "      <td>560.9</td>\n",
       "      <td>-0.29</td>\n",
       "      <td>-0.10</td>\n",
       "      <td>-1.11</td>\n",
       "      <td>124.3</td>\n",
       "      <td>791.6</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.39</td>\n",
       "      <td>-1.85</td>\n",
       "      <td>-21.7</td>\n",
       "      <td>-44.9</td>\n",
       "      <td>-0.21</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.89</td>\n",
       "      <td>133.9</td>\n",
       "      <td>797.8</td>\n",
       "      <td>-0.08</td>\n",
       "      <td>1.06</td>\n",
       "      <td>-0.26</td>\n",
       "      <td>-16.4</td>\n",
       "      <td>-74.1</td>\n",
       "      <td>0.97</td>\n",
       "      <td>-0.80</td>\n",
       "      <td>-0.41</td>\n",
       "      <td>66.9</td>\n",
       "      <td>955.3</td>\n",
       "      <td>-1.90</td>\n",
       "      <td>1.28</td>\n",
       "      <td>-6.65</td>\n",
       "      <td>-28.1</td>\n",
       "      <td>47.5</td>\n",
       "      <td>-1.91</td>\n",
       "      <td>1.42</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-30.0</td>\n",
       "      <td>1846.7</td>\n",
       "      <td>0.76</td>\n",
       "      <td>1.10</td>\n",
       "      <td>-4.0</td>\n",
       "      <td>-52.0</td>\n",
       "      <td>-53.9</td>\n",
       "      <td>1.71</td>\n",
       "      <td>-0.22</td>\n",
       "      <td>-12.0</td>\n",
       "      <td>97.5</td>\n",
       "      <td>1969.8</td>\n",
       "      <td>-1.70</td>\n",
       "      <td>0.16</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>-32.5</td>\n",
       "      <td>255.9</td>\n",
       "      <td>-0.46</td>\n",
       "      <td>1.57</td>\n",
       "      <td>10.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2047.7</td>\n",
       "      <td>-0.98</td>\n",
       "      <td>1.53</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-49.0</td>\n",
       "      <td>554.2</td>\n",
       "      <td>-0.83</td>\n",
       "      <td>0.39</td>\n",
       "      <td>0.73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>279</td>\n",
       "      <td>261536</td>\n",
       "      <td>0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>28.00</td>\n",
       "      <td>-1.32</td>\n",
       "      <td>-9.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>464.8</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.19</td>\n",
       "      <td>8.0</td>\n",
       "      <td>-51.5</td>\n",
       "      <td>98.1</td>\n",
       "      <td>1.09</td>\n",
       "      <td>-0.33</td>\n",
       "      <td>-2.16</td>\n",
       "      <td>-3.9</td>\n",
       "      <td>102.7</td>\n",
       "      <td>0.39</td>\n",
       "      <td>-1.22</td>\n",
       "      <td>-3.39</td>\n",
       "      <td>-15.2</td>\n",
       "      <td>-42.2</td>\n",
       "      <td>-1.18</td>\n",
       "      <td>-1.11</td>\n",
       "      <td>-3.55</td>\n",
       "      <td>8.9</td>\n",
       "      <td>141.3</td>\n",
       "      <td>-0.16</td>\n",
       "      <td>-0.43</td>\n",
       "      <td>-4.15</td>\n",
       "      <td>-12.9</td>\n",
       "      <td>-13.4</td>\n",
       "      <td>-1.32</td>\n",
       "      <td>-0.98</td>\n",
       "      <td>-3.69</td>\n",
       "      <td>8.8</td>\n",
       "      <td>136.1</td>\n",
       "      <td>-0.30</td>\n",
       "      <td>4.13</td>\n",
       "      <td>1.89</td>\n",
       "      <td>-13.0</td>\n",
       "      <td>-18.7</td>\n",
       "      <td>-1.37</td>\n",
       "      <td>-0.93</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>810.1</td>\n",
       "      <td>-2.29</td>\n",
       "      <td>6.72</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-23.0</td>\n",
       "      <td>-29.7</td>\n",
       "      <td>0.58</td>\n",
       "      <td>-1.10</td>\n",
       "      <td>-18.5</td>\n",
       "      <td>33.5</td>\n",
       "      <td>206.8</td>\n",
       "      <td>1.84</td>\n",
       "      <td>-0.13</td>\n",
       "      <td>4.0</td>\n",
       "      <td>-29.0</td>\n",
       "      <td>30.1</td>\n",
       "      <td>0.80</td>\n",
       "      <td>-0.24</td>\n",
       "      <td>5.0</td>\n",
       "      <td>-14.0</td>\n",
       "      <td>479.5</td>\n",
       "      <td>0.68</td>\n",
       "      <td>-0.59</td>\n",
       "      <td>2.0</td>\n",
       "      <td>-36.0</td>\n",
       "      <td>-6.9</td>\n",
       "      <td>2.02</td>\n",
       "      <td>0.14</td>\n",
       "      <td>-0.23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   EXAMPLE_ID  BLOCK_ID  target     4      5     6     7     8       9    10  \\\n",
       "0         279    261532       0  52.0  32.69  0.30   2.5  20.0  1256.8 -0.89   \n",
       "1         279    261533       0  58.0  33.33  0.00  16.5   9.5   608.1  0.50   \n",
       "2         279    261534       0  77.0  27.27 -0.91   6.0  58.5  1623.6 -1.40   \n",
       "3         279    261535       0  41.0  27.91 -0.35   3.0  46.0  1921.6 -1.36   \n",
       "4         279    261536       0  50.0  28.00 -1.32  -9.0  12.0   464.8  0.88   \n",
       "\n",
       "     11    12    13     14    15    16    17     18     19    20    21    22  \\\n",
       "0  0.33  11.0 -55.0  267.2  0.52  0.05 -2.36   49.6  252.0  0.43  1.16 -2.06   \n",
       "1  0.07  20.5 -52.5  521.6 -1.08  0.58 -0.02   -3.2  103.6 -0.95  0.23 -2.87   \n",
       "2  0.02  -6.5 -48.0  621.0 -1.20  0.14 -0.20   73.6  609.1 -0.44 -0.58 -0.04   \n",
       "3 -0.47 -32.0 -51.5  560.9 -0.29 -0.10 -1.11  124.3  791.6  0.00  0.39 -1.85   \n",
       "4  0.19   8.0 -51.5   98.1  1.09 -0.33 -2.16   -3.9  102.7  0.39 -1.22 -3.39   \n",
       "\n",
       "     23     24    25    26    27     28     29    30    31    32    33     34  \\\n",
       "0 -33.0 -123.2  1.60 -0.49 -6.06   65.0  296.1 -0.28 -0.26 -3.83 -22.6 -170.0   \n",
       "1 -25.9  -52.2 -0.21  0.87 -1.81   10.4   62.0 -0.28 -0.04  1.48 -17.6 -198.3   \n",
       "2 -23.0  -27.4 -0.72 -1.04 -1.09   91.1  635.6 -0.88  0.24  0.59 -18.7   -7.2   \n",
       "3 -21.7  -44.9 -0.21  0.02  0.89  133.9  797.8 -0.08  1.06 -0.26 -16.4  -74.1   \n",
       "4 -15.2  -42.2 -1.18 -1.11 -3.55    8.9  141.3 -0.16 -0.43 -4.15 -12.9  -13.4   \n",
       "\n",
       "     35    36    37    38     39    40    41    42    43     44    45    46  \\\n",
       "0  3.06 -1.05 -3.29  22.9  286.3  0.12  2.58  4.08 -33.0 -178.9  1.88  0.53   \n",
       "1  3.43  2.84  5.87 -16.9   72.6 -0.31  2.79  2.71 -33.5  -11.6 -1.11  4.01   \n",
       "2 -0.60 -2.82 -0.71  52.4  504.1  0.89 -0.67 -9.30 -20.8  -25.7 -0.77 -0.85   \n",
       "3  0.97 -0.80 -0.41  66.9  955.3 -1.90  1.28 -6.65 -28.1   47.5 -1.91  1.42   \n",
       "4 -1.32 -0.98 -3.69   8.8  136.1 -0.30  4.13  1.89 -13.0  -18.7 -1.37 -0.93   \n",
       "\n",
       "    47    48      49    50    51   52    53     54    55    56    57     58  \\\n",
       "0 -7.0 -44.0  1987.0 -5.41  0.95 -4.0 -57.0  722.9 -3.26 -0.55  -7.5  125.5   \n",
       "1  5.0 -57.0   666.3  1.13  4.38  5.0 -64.0   39.3  1.07 -0.16  32.5  100.0   \n",
       "2  0.0 -20.0  2259.0 -0.94  1.15 -4.0 -44.0  -22.7  0.94 -0.98 -19.0  105.0   \n",
       "3  1.0 -30.0  1846.7  0.76  1.10 -4.0 -52.0  -53.9  1.71 -0.22 -12.0   97.5   \n",
       "4  0.0  -1.0   810.1 -2.29  6.72  1.0 -23.0  -29.7  0.58 -1.10 -18.5   33.5   \n",
       "\n",
       "       59    60    61    62    63     64    65    66    67    68      69  \\\n",
       "0  1547.2 -0.36  1.12   9.0 -37.0   72.5  0.47  0.74 -11.0  -8.0  1595.1   \n",
       "1  1893.7 -2.80 -0.22   2.5 -28.5   45.0  0.58  0.41 -19.0  -6.0   762.9   \n",
       "2  1267.9  1.03  1.27  11.0 -39.5   82.3  0.47 -0.19 -10.0   7.0  1491.8   \n",
       "3  1969.8 -1.70  0.16  -1.0 -32.5  255.9 -0.46  1.57  10.0   6.0  2047.7   \n",
       "4   206.8  1.84 -0.13   4.0 -29.0   30.1  0.80 -0.24   5.0 -14.0   479.5   \n",
       "\n",
       "     70    71   72    73     74    75    76    77  \n",
       "0 -1.64  2.83 -2.0 -50.0  445.2 -0.35  0.26  0.76  \n",
       "1  0.29  0.82 -3.0 -35.0  140.3  1.16  0.39  0.73  \n",
       "2  0.32 -1.29  0.0 -34.0  658.2 -0.76  0.26  0.24  \n",
       "3 -0.98  1.53  0.0 -49.0  554.2 -0.83  0.39  0.73  \n",
       "4  0.68 -0.59  2.0 -36.0   -6.9  2.02  0.14 -0.23  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = load_bio_kdd04(as_frame=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "eb1768e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "target\n",
       "0    144455\n",
       "1      1296\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# imbalance of the classes\n",
    "df[\"target\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "141f0e67",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop columns we won't need in this example\n",
    "df.drop(columns=[\"EXAMPLE_ID\", \"BLOCK_ID\"], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8159efde",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train, df_valid = train_test_split(\n",
    "    df, test_size=0.2, stratify=df[\"target\"], random_state=1\n",
    ")\n",
    "df_valid, df_test = train_test_split(\n",
    "    df_valid, test_size=0.5, stratify=df_valid[\"target\"], random_state=1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8b6d9f0",
   "metadata": {},
   "source": [
    "## Preparing the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d9bcf02a",
   "metadata": {},
   "outputs": [],
   "source": [
    "continuous_cols = df.drop(columns=[\"target\"]).columns.values.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "3618ceb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# deeptabular\n",
    "tab_preprocessor = TabPreprocessor(continuous_cols=continuous_cols, scale=True)\n",
    "X_tab_train = tab_preprocessor.fit_transform(df_train)\n",
    "X_tab_valid = tab_preprocessor.transform(df_valid)\n",
    "X_tab_test = tab_preprocessor.transform(df_test)\n",
    "\n",
    "# target\n",
    "y_train = df_train[\"target\"].values\n",
    "y_valid = df_valid[\"target\"].values\n",
    "y_test = df_test[\"target\"].values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f270bf8",
   "metadata": {},
   "source": [
    "## Define the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5781ea4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_layer = len(tab_preprocessor.continuous_cols)\n",
    "output_layer = 1\n",
    "hidden_layers = np.linspace(\n",
    "    input_layer * 2, output_layer, 5, endpoint=False, dtype=int\n",
    ").tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "9d7dcc60",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "WideDeep(\n",
       "  (deeptabular): Sequential(\n",
       "    (0): TabMlp(\n",
       "      (cont_norm): Identity()\n",
       "      (encoder): MLP(\n",
       "        (mlp): Sequential(\n",
       "          (dense_layer_0): Sequential(\n",
       "            (0): Linear(in_features=74, out_features=148, bias=True)\n",
       "            (1): ReLU(inplace=True)\n",
       "            (2): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (dense_layer_1): Sequential(\n",
       "            (0): Linear(in_features=148, out_features=118, bias=True)\n",
       "            (1): ReLU(inplace=True)\n",
       "            (2): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (dense_layer_2): Sequential(\n",
       "            (0): Linear(in_features=118, out_features=89, bias=True)\n",
       "            (1): ReLU(inplace=True)\n",
       "            (2): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (dense_layer_3): Sequential(\n",
       "            (0): Linear(in_features=89, out_features=59, bias=True)\n",
       "            (1): ReLU(inplace=True)\n",
       "            (2): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "          (dense_layer_4): Sequential(\n",
       "            (0): Linear(in_features=59, out_features=30, bias=True)\n",
       "            (1): ReLU(inplace=True)\n",
       "            (2): Dropout(p=0.1, inplace=False)\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "    (1): Linear(in_features=30, out_features=1, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deeptabular = TabMlp(\n",
    "    mlp_hidden_dims=hidden_layers,\n",
    "    column_idx=tab_preprocessor.column_idx,\n",
    "    continuous_cols=tab_preprocessor.continuous_cols,\n",
    ")\n",
    "model = WideDeep(deeptabular=deeptabular)\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "e8293c96",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Metrics from torchmetrics\n",
    "accuracy = Accuracy_torchmetrics(average=None, num_classes=1, task=\"binary\")\n",
    "precision = Precision_torchmetrics(average=\"micro\", num_classes=1, task=\"binary\")\n",
    "f1 = F1_torchmetrics(average=None, num_classes=1, task=\"binary\")\n",
    "recall = Recall_torchmetrics(average=None, num_classes=1, task=\"binary\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1bd7147-31f6-47da-bfae-f6194c4f25bd",
   "metadata": {},
   "source": [
    "**Note**:\n",
    "\n",
    "Following cells includes usage of both `RayTuneReporter` and `WnBReportBest` callbacks.\n",
    "In case you want to use just `RayTuneReporter`, remove following:\n",
    "* wandb from config\n",
    "* `WandbLoggerCallback`\n",
    "* `WnBReportBest`\n",
    "* `@wandb_mixin` decorator\n",
    "\n",
    "We do not see strong reason to use WnB without RayTune for a single paramater combination run, but it is possible:\n",
    "* **option01**: define paramaters in config only for a single value `tune.grid_search([1000])` (single value RayTune run)\n",
    "* **option02**: define WnB callback that reports currnet validation/training loss, metrics, etc. at the end of batch, ie. do not report to WnB at `epoch_end` as in `WnBReportBest` but at the `on_batch_end`, see `pytorch_widedeep.callbacks.Callback`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b841d015",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/javierrodriguezzaurin/.pyenv/versions/3.10.13/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/_2/lrjn1qn54c758tdtktr1bvkc0000gn/T/tmp60pfyl1kwandb'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/javierrodriguezzaurin/.pyenv/versions/3.10.13/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/_2/lrjn1qn54c758tdtktr1bvkc0000gn/T/tmpnjv2rg1wwandb-artifacts'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/javierrodriguezzaurin/.pyenv/versions/3.10.13/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/_2/lrjn1qn54c758tdtktr1bvkc0000gn/T/tmpgebu5k1kwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n",
      "/Users/javierrodriguezzaurin/.pyenv/versions/3.10.13/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/_2/lrjn1qn54c758tdtktr1bvkc0000gn/T/tmpxy9y2yriwandb-media'>\n",
      "  _warnings.warn(warn_message, ResourceWarning)\n"
     ]
    }
   ],
   "source": [
    "config = {\n",
    "    \"batch_size\": tune.grid_search([1000, 5000]),\n",
    "    \"wandb\": {\n",
    "        \"project\": \"test\",\n",
    "        # \"api_key_file\": os.getcwd() + \"/wandb_api.key\",\n",
    "        \"api_key\": \"WNB_API_KEY\",\n",
    "    },\n",
    "}\n",
    "\n",
    "# Optimizers\n",
    "deep_opt = SGD(model.deeptabular.parameters(), lr=0.1)\n",
    "# LR Scheduler\n",
    "deep_sch = lr_scheduler.StepLR(deep_opt, step_size=3)\n",
    "\n",
    "\n",
    "@wandb_mixin\n",
    "def training_function(config, X_train, X_val):\n",
    "    early_stopping = EarlyStopping()\n",
    "    model_checkpoint = ModelCheckpoint(save_best_only=True)\n",
    "    # Hyperparameters\n",
    "    batch_size = config[\"batch_size\"]\n",
    "    trainer = Trainer(\n",
    "        model,\n",
    "        objective=\"binary_focal_loss\",\n",
    "        callbacks=[\n",
    "            RayTuneReporter,\n",
    "            WnBReportBest(wb=wandb),\n",
    "            early_stopping,\n",
    "            model_checkpoint,\n",
    "        ],\n",
    "        lr_schedulers={\"deeptabular\": deep_sch},\n",
    "        initializers={\"deeptabular\": XavierNormal},\n",
    "        optimizers={\"deeptabular\": deep_opt},\n",
    "        metrics=[accuracy, precision, recall, f1],\n",
    "        verbose=0,\n",
    "    )\n",
    "\n",
    "    trainer.fit(X_train=X_train, X_val=X_val, n_epochs=5, batch_size=batch_size)\n",
    "\n",
    "\n",
    "X_train = {\"X_tab\": X_tab_train, \"target\": y_train}\n",
    "X_val = {\"X_tab\": X_tab_valid, \"target\": y_valid}\n",
    "\n",
    "asha_scheduler = AsyncHyperBandScheduler(\n",
    "    time_attr=\"training_iteration\",\n",
    "    metric=\"_metric/val_loss\",\n",
    "    mode=\"min\",\n",
    "    max_t=100,\n",
    "    grace_period=10,\n",
    "    reduction_factor=3,\n",
    "    brackets=1,\n",
    ")\n",
    "\n",
    "analysis = tune.run(\n",
    "    tune.with_parameters(training_function, X_train=X_train, X_val=X_val),\n",
    "    resources_per_trial={\"cpu\": 1, \"gpu\": 0},\n",
    "    progress_reporter=JupyterNotebookReporter(overwrite=True),\n",
    "    scheduler=asha_scheduler,\n",
    "    config=config,\n",
    "    callbacks=[\n",
    "        WandbLoggerCallback(\n",
    "            project=config[\"wandb\"][\"project\"],\n",
    "            # api_key_file=config[\"wandb\"][\"api_key_file\"],\n",
    "            api_key=config[\"wandb\"][\"api_key\"],\n",
    "            log_config=True,\n",
    "        )\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "81d581da",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'fc9a8_00000': {'_metric': {'train_loss': 0.006297602537127896,\n",
       "   'train_Accuracy': 0.9925042986869812,\n",
       "   'train_Precision': 0.9939393997192383,\n",
       "   'train_Recall': 0.15814851224422455,\n",
       "   'train_F1Score': 0.2728785574436188,\n",
       "   'val_loss': 0.005045663565397263,\n",
       "   'val_Accuracy': 0.9946483969688416,\n",
       "   'val_Precision': 1.0,\n",
       "   'val_Recall': 0.39534884691238403,\n",
       "   'val_F1Score': 0.5666667222976685},\n",
       "  'time_this_iter_s': 2.388202428817749,\n",
       "  'done': True,\n",
       "  'timesteps_total': None,\n",
       "  'episodes_total': None,\n",
       "  'training_iteration': 5,\n",
       "  'trial_id': 'fc9a8_00000',\n",
       "  'experiment_id': 'baad1d4f3d924b48b9ece1b9f26c80cc',\n",
       "  'date': '2022-07-31_14-06-51',\n",
       "  'timestamp': 1659276411,\n",
       "  'time_total_s': 12.656474113464355,\n",
       "  'pid': 1813,\n",
       "  'hostname': 'jupyter-5uperpalo',\n",
       "  'node_ip': '10.32.44.172',\n",
       "  'config': {'batch_size': 1000},\n",
       "  'time_since_restore': 12.656474113464355,\n",
       "  'timesteps_since_restore': 0,\n",
       "  'iterations_since_restore': 5,\n",
       "  'warmup_time': 0.8006253242492676,\n",
       "  'experiment_tag': '0_batch_size=1000'},\n",
       " 'fc9a8_00001': {'_metric': {'train_loss': 0.02519632239515583,\n",
       "   'train_Accuracy': 0.9910891652107239,\n",
       "   'train_Precision': 0.25,\n",
       "   'train_Recall': 0.0009643201483413577,\n",
       "   'train_F1Score': 0.0019212296465411782,\n",
       "   'val_loss': 0.02578434906899929,\n",
       "   'val_Accuracy': 0.9911492466926575,\n",
       "   'val_Precision': 0.0,\n",
       "   'val_Recall': 0.0,\n",
       "   'val_F1Score': 0.0},\n",
       "  'time_this_iter_s': 4.113586902618408,\n",
       "  'done': True,\n",
       "  'timesteps_total': None,\n",
       "  'episodes_total': None,\n",
       "  'training_iteration': 5,\n",
       "  'trial_id': 'fc9a8_00001',\n",
       "  'experiment_id': 'f2e54a6a5780429fbf0db0746853347e',\n",
       "  'date': '2022-07-31_14-06-56',\n",
       "  'timestamp': 1659276416,\n",
       "  'time_total_s': 12.926990509033203,\n",
       "  'pid': 1962,\n",
       "  'hostname': 'jupyter-5uperpalo',\n",
       "  'node_ip': '10.32.44.172',\n",
       "  'config': {'batch_size': 5000},\n",
       "  'time_since_restore': 12.926990509033203,\n",
       "  'timesteps_since_restore': 0,\n",
       "  'iterations_since_restore': 5,\n",
       "  'warmup_time': 0.9253025054931641,\n",
       "  'experiment_tag': '1_batch_size=5000'}}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analysis.results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56311243",
   "metadata": {},
   "source": [
    "Using Weights and Biases logging you can create [parallel coordinates graphs](https://docs.wandb.ai/ref/app/features/panels/parallel-coordinates) that map parametr combinations to the best(lowest) loss achieved during the training of the networks\n",
    "\n",
    "![WNB](figures/wnb.png \"parallel coordinates\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b469bc2",
   "metadata": {},
   "source": [
    "local visualization of raytune reults using tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "57b44a55",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%load_ext tensorboard\n",
    "%tensorboard --logdir ~/ray_results"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "vscode": {
   "interpreter": {
    "hash": "bee110fa72fc220f84be99700c69baf478c6696e63cfda5b1944123ebc470d26"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
